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(54) PICTURE EXTRACTION DEVICE 

(57)Abstract: 

PROBLEM TO BE SOLVED: To easily and automatically 
segment a desired object from an inputted picture. 
SOLUTION: A first feature extraction means 2 extracts 
the motion vector of the inputted picture, and a feature 
homogeneity evaluation means 3 and an area dividing 
means 5 set the distribution of the dimension and the 
direction of the motion vector to be an evaluation value 
to divide the area of the evaluation value within a 
prescribed limit. An area growing means 7 extracts the 
area of a color specified as a second feature from this 
divided area and connects the area, and a display means 
8 displays it. A divided picture means 6 displays the 
divided area as a label picture. Thereby the object is 
simply and automatically segmented only by roughly 
specifying the position of the desired area based on the 
label picture by means of a mouse, etc. on a displayed 
picture. 
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JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] The image extractor had a primary feature-extraction means extract the primary 
description of input image data, a field division means divide the field which has the above- 
mentioned primary description, a field selection means choose the field which fulfills 
predetermined conditions from the division field by which division was carried out [ above- 
mentioned ] ? a field growth means grow up the field by using as a nucleus the field by which 
selection was made [ above-mentioned ], and an image-display means display the image after 
the above-mentioned field growth. 

[Claim 2] It is the image extractor according to claim 1 which carries out [ that the above- 
mentioned field growth means consists of a secondary feature-extraction means extract the 
secondary description from the field by which selection was made / above-mentioned /, a 
growth nucleus setting means set up the nucleus for field growth based on the above-mentioned 
secondary description, a secondary description field growth means to by_which the secondary 
description of a proper performs field growth in the above-mentioned nucleus, and a connection 
means connect two or more fields after field growth, and ] as the description. 
[Claim 3] the above-mentioned field division means — distribution of the above-mentioned 
primary description data — abbreviation — the image extractor according to claim 1 
characterized by performing division which carries out grouping of the homogeneous field. 
[Claim 4] The above-mentioned field division means is an image extractor according to claim 1 
characterized by making the field more than predetermined size into the above-mentioned 
division field among the connection fields obtained by carrying out connection processing of the 
field which has the above-mentioned primary description. 

[Claim 5] It is the image extractor according to claim 1 which the above-mentioned field division 
means has the means which carries out label attachment of each of the above-mentioned 
division field, establishes an image generation means to generate the label image data which has 
the predetermined description, all over the same field of the above-mentioned label, and is 
characterized by for the above-mentioned image display means to display the above-mentioned 
label image data and the above-mentioned input image data in piles. 

[Claim 6] It is the image extractor according to claim 1 which the above-mentioned field 
selection means chooses what has the maximum area among the above-mentioned division 
fields, and is characterized by the above-mentioned image display means displaying the image 
data of the field by which selection was made [ above-mentioned ]. 

[Claim 7] It is the image extractor according to claim 1 which the above-mentioned field 
selection means chooses what has average brightness or an average hue in the predetermined 
range among the above-mentioned division fields, and is characterized by the above-mentioned 
image display means displaying the image data of the field by which selection was made [ above- 
mentioned ]. 

[Claim 8] It is the image extractor according to claim 1 which the above-mentioned input image 
data is time series image data, and is characterized by the above-mentioned primary description 
being motion vector distribution. 

[Claim 9] It is the image extractor according to claim 1 which the above-mentioned input image 
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data is time series image data, and is characterized by the above-mentioned primary description 
being subtraction-image distribution of input image data. 

[Claim 10] The above-mentioned input image data is an image extractor according to claim 1 
which consists of two or more image data from which the view location in the same scene 
differs, and is characterized by the above-mentioned primary description being parallax 
distribution between two or more above-mentioned image data. 

[Claim 11] The above-mentioned field selection means is the image extractor according to claim 
5 carry out having a marker generation means for displaying a marker on the location which 
directed with the above-mentioned field selection directions means based on the above- 
mentioned label image displayed on the field selection directions means and the above- 
mentioned image-display means on the above-mentioned image-display means, and a judgment 
means judge a field including the above-mentioned marker position to be a selection field as the 
description. 

[Claim 1 2] The above-mentioned input image data is an image extractor according to claim 1 
obtained from an image pick-up means. 

[Claim 13] The above-mentioned image pick-up means is an image extractor according to claim 
12 which is what is prepared in the mobile. 

[Claim 14] The above-mentioned image pick-up means is an image extractor according to claim 
12 which is what has a compound eye image pick-up system. 

[Claim 15] The image extractor according to claim 14 whose above-mentioned primary 
description is parallax distribution of the corresponding points between each image pick-up 
system in the above-mentioned compound eye image pick-up system. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the image extractor which performs logging of 

an image and the extract of a border line. 

[0002] 

[Description of the Prior Art] Conventionally, the approach of placing the object as an approach 
of starting the field of a desired object into the background which has a fixed color (blue), and 
carrying out chroma-key processing from an image, is learned well. Moreover, the range started 
along with the border line of an object is specified by the help, and the methods (JP, 6-251 149,A, 
JP,7-37086,A, etc.) of asking a key for information, such as a color component and brightness, 
for the exact location of logging by the image processing etc. are learned. As an example which 
attained a certain amount of automation, an operator specifies one point and the profile trace 
direction of [ on a border line ] indicated by JP, 7-361 73,A, and there is the approach of tracking 
a profile automatically based on the inclination data of shade distribution of the image obtained 
by differential processing. This is the image logging approach which interrupted processing, 
inputs the trace direction etc. anew based on directions of an operator when a profile branches, 
or when ambiguous, and was equipped with motility and interactive nature. 
[0003] Moreover, in image-processing tools, such as Photoshop (Adobe company), if one in a 
field to extract is specified with a mouse in the case of logging from a background, including the 
point, a color component can connect the field in predetermined within the limits automatically, 
and the field of different color component range can be similarly formed and merged. As an 
approach of further others, the background image is memorized and there are methods (JP.4- 
21 61 81, A, JP,6-266841,A, JP,6-268902,A, etc.) of extracting a quiescence body image from a 
subtraction image with the present image. 
[0004] 

[Problem(s) to be Solved by the Invention] However, the approach by the chroma-key 
processing mentioned above had problems, like the part is missing at the time of an image 
extract, when it had that a background must be a specific color and the color as a background 
with the same object. Moreover, the method of specifying a border line by the operator 
essentially extracts most profile configurations of an outline by the help, and the extract from 
the image accompanied by the body of a complicated configuration, a motion, or deformation was 
difficult. Moreover, the approach of pursuing this border line had the indispensable management 
by the operator, when an object had a complicated pattern, and when it had further a background 
and texture (pattern) structure to which the object was similar and processing by the help took 
place frequently, it had the problem that the semantics of automation was lost. 
[0005] By the approach of carrying out automatic extracting of the field where the color 
component approached using the image-processing tool, there was a problem that it was 
generally impossible to cut down an object image by one actuation, and the field merge covering 
many times was needed. Furthermore, it was difficult stability and to not necessarily start only a 
target object correctly according to factors, such as a noise, by the approach using difference 
with a background image, the image obtained while performing camera actuation of panning etc. 
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when especially a camera was used as an image input means — setting — difference — it could 
not respond only by processing but there was a problem that increase of the amount of 
operations — global matching processing is needed — was not avoided. 

[0006] Then, the 1st purpose of this invention is only directing the location of an object roughly 
and making automatic extracting of an image possible, without pursuing the border line by the 
help or the image processing. The 2nd purpose of this invention is performing efficiently 
narrowing down of an object image, and the minute extract from a background by using the 
classification of two or more description data. 

[0007] the field where the 3rd purpose of this invention was divided based on the field which had 
complete set of description data roughly — as the nucleus candidate field of a field grown 
method — directions — it is supposing that it is selectable and making automatic extracting of a 
specific image possible, the field divided based on little data, without the 4th purpose of this 
invention evaluating whenever [ homogeneity / of the description ] — as the nucleus candidate 
field of a field grown method — directions — it is supposing that it is selectable and making 
automatic extracting of a specific image possible. 

[0008] The 5th purpose of this invention is making easy the selection directions for [ by the 
operator ] an extract by displaying in piles the field divided roughly on an input image. The 6th 
purpose of this invention is it carrying out automatic extracting, using as the candidate for an 
extract the maximum area field which has the same image attribute, and realizing simplification 
and improvement in the speed of processing. The 7th purpose of this invention is an average 
brightness and color component's using the image field in the predetermined range as a field 
growth nucleus, carrying out automatic extracting, and realizing the simplification and 
improvement in the speed of processing. The 8th purpose of this invention is realizing the image 
extract based on the 1st purpose at the time of extracting a subject of operation from a time 
series image. 

[0009] The 9th purpose of this invention is realizing the image extract based on the 1st purpose 
at the time of extracting a subject of operation from a time series image at a high speed. The 
10th purpose of this invention is realizing the image extract based on the 1st purpose, when the 
average distance of the for a background and for an extract from a view location differs enough. 
The 1 1th purpose of this invention is realizing the image extract based on the 1st purpose by 
rough directions selection interactively. 
[0010] 

[Means for Solving the Problem] A primary feature-extraction means extract the primary 
description of input image data in this invention, a field division means divide the field which has 
the above-mentioned primary description, a field selection means choose the field which fulfills 
predetermined conditions from the division field by which division was carried out [ above- 
mentioned ], a field growth means grow up the field by using as a nucleus the field by which 
selection was made [ above-mentioned ], and an image-display means display the image after 
the above-mentioned field growth have prepared. 
[0011] 

[Function] According to this invention, the field which has the primary description first from 
input image data is divided, and the field which fulfills further predetermined conditions among 
the divided field is chosen, and, for example based on the secondary description, expansion 
growth of the field is further carried out from the selected field, it goes, and the image after 
growth is displayed. Therefore, only by carrying out tab control specification of the field which 
should be extracted on the displayed image roughly, an object image can be cut down 
automatically easily (extract) and can be carried out. 
[0012] 

[Embodiment of the Invention] Drawing 1 is drawing showing the example of a system 
configuration of this invention as a gestalt of the 1st operation notionally. In drawing, 1 is an 
image input means and an image pick-up means or an image database is used. For the field 
division means based on [ 2 / 3 / the primary description data extraction means in input image 
data, and / whenever / homogeneity / of the description data / based on the image storage 
means of an input image in an evaluation means and 4 ] whenever [ description homogeneity ] in 



http://www4.ipdl.ncipi.go.jp/cgi-bin/tran_web_cgi_ejje 



2007/03/20 



THIS PAGE BUNK (uspto; 



JP.09-185720.A [DETAILED DESCRIPTION] 



3/6 ^— v 



5, and 6, as for a field growth means and 8 f a division image generation means and 7 are [ a 
display means and 9 ] the selection directions means of a division field, and pointing means, such 
as a mouse, are used. 

[0013] In addition, field growth is evaluating whenever [ homogeneity / of the characteristic 
quantity in the predetermined field in an image ], and performing division of a field, and a merge. 
While the description of the gestalt of this operation extracts roughly the image field which has 
extract possibility by extracting the primary description data in an input image at a high speed, 
and performing field division of an image based on the distribution It is performing field growth 
based on the secondary characteristic quantity (characteristic quantity from which the primary 
description, classification, an attribute, etc. differ) which uses a candidate field as a nucleus as a 
following step for a more minute extract. In the field extracted based on the primary description 
data, image information required to perform field growth based on the secondary description shall 
be contained. 

[0014] In case the primary description data extraction means 2 extracts the field used as the 
nucleus at the time of performing processing by field growth means 7 to explain later at a high 
speed and actually extracts the image field for specification, the rough selection and the 
directions by the operator can perform it. Here, it explains as what extracts the motion vector in 
screen top each point as an example of the primary description data. 

[0015] The distribution about the magnitude and the direction of a motion vector of [ in a 
predetermined field ] is used for the evaluation means 3 as an evaluation value whenever 
[ description homogeneity ]. Compaction and the increase in efficiency of the processing time 
are attained by setting up the size of the field (block) to evaluate appropriately based on the 
target size in a screen. That is, as compared with the target size in every direction [ in a 
screen ], about several [ 1/] to 1/10 division field extent is used typically. You may make it set 
up this value suitably beforehand by the operator side. 

[0016] If the field division means 5 has the evaluation value of whenever [ above-mentioned 
homogeneity ] smaller than a predetermined threshold, it will consider that the field is 
homogeneous and will connect the fields which have an evaluation value in predetermined 
tolerance whenever [ homogeneity ]. Moreover, the field outside tolerance forms the lump 
(plurality is good) of a field which can dissociate and can represent with one motion vector. The 
input image picturized by drawing 7 (a) with the fixed camera is shown, and the result of the field 
division performed based on the motion vector is shown in this drawing (b). The result of image 
logging by the field growth based on the secondary descriptions, such as a color component, is 
shown in this drawing (c). 

[0017] In addition, the field after division may not necessarily reflect the configuration of an 
actual object in a precision. Therefore, although the primary feature-extraction means 2 may be 
somewhat inferior to the precision (resolution) of the secondary feature-extraction means 71 
used later, to be able to extract at a high speed is desired. However, in case it is that some 
background images are not intermingled in a division field, and field growth, it is desirable to 
include all required image attributes (secondary description). For the former, it is good after field 
division also as a division field used for the processing which follows what reduced or deleted 
each field at a fixed rate. 

[0018] The division image generation means 6 gives the image data (for example, a different 
color, a different hatching pattern, etc.) of an attribute which is different corresponding to two or 
more fields divided based on the primary description data, respectively, and it is displayed that it 
laps with input image data on the display means 8. Label attachment is performed to the 
specifically divided field, and it processes smearing away the inside of a mask by the fixed 
pattern etc. by using the same label field as mask data. By doing in this way, an operator can 
distinguish easily the candidate field which should be directed by viewing, and assignment for 
[ by the division field directions means 9 ] an extract becomes certain (since it is beforehand 
classified with the same label), and easy. Although a mouse etc. is typically used as a division 
field directions means 9, especially when using a touch panel mold display, there is no need. 
[0019] The field growth means 7 is divided based on the primary description data as mentioned 
above. The secondary description (attribute) from the selected field It consists of a growth 
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nucleus setting means 72 in the case of performing field growth based on a secondary feature- 
extraction means 71 to extract, and its secondary description, a field growth means 73 to 
perform field growth for the secondary description of a proper based on each growth nucleus, 
and a field connection means 74 to connect each field after growth. In addition, field growth 
means performing division of a field, and a merge according to whenever [ homogeneity / of the 
description data ], and, on the other hand, merge processing is not necessarily given to a target. 
Moreover, although there are some which have the thing which has geometric structure usually, 
for example or luminance distribution, and the high order description which processes a color 
component and is acquired that the secondary description should just be a fundamentally 
different thing from the primary description, it is not necessarily limited to these, for example, 
color component information etc. is used suitably. However, it is satisfactory, if the minute image 
division by field growth is possible even if it is the same classification (here motion vector) as 
the primary description (refer to the gestalt of the 5th operation). 

[0020] After extracting the secondary description as mentioned above, the initial field of a 
proper is set as each with the growth nucleus setting means 72. When secondary characterized 
by color component information, the secondary description plurality should grow up to be may 
exist in the extracted candidate field. For example, in the case of a person's image, it has a hue 
which is different on the average according to parts, such as a face, the hair, and clothes, 
respectively, but it considers as the profile part which has the secondary description from which 
the candidate field which was extracted as a growth nucleus in that case based on the primary 
description data, and was chosen with the division field directions means 9 differs, or its near 
field. After equalizing within the near field block of predetermined size beforehand in the case of 
field growth, you may make it grow up. The effect of fluctuation of the image data (attribute) in 
the resistance and pixel size order to a noise can be eased by this, and extract precision can be 
raised. 

[0021] Drawing 2 shows the gestalt of the 2nd operation and shows the means 2-9 which 
materialized other notional means 2-9 of drawing 1 at the time of using the image pick-up means 
1 for the image input means 1 of drawing 1 . It is the case where made the primary description 
data into the motion vector here, and the secondary description data are made into a hue. In 
addition, about the detection algorithm of a motion vector, although explanation is omitted since 
it is not the chief aim of this invention, especially approaches with rapidity as shown in a "robot 
vision" (the Yanaida work, Shokodo issue), such as a gradient method and a correlation 
technique, are desired, for example. 

[0022] It may dissociate from the background for specification based on the difference of a 
motion vector in case all that include actuation, the separation from the quiescence background 
of the object accompanied by a motion, or a background as an application example of the gestalt 
of this operation move, for example, — the case where a certain scene is picturized while the 
image pick-up means 1 is installed in the body, a robot, or a car body and the image pick-up 

means 1 moves — general — the screen whole — crossing — un zero — a motion vector 

exists. In such a case, if a background and the distance from the image pick-up means 1 of an 
object are especially separated enough, it can move like the above-mentioned and field division 
based on a vector can be performed comparatively with high precision. When there are few 
objects which carry out the object which moves especially, or a different motion from a 
perimeter, the gestalt of this operation is effective. 

[0023] Drawing 3 shows the gestalt of the 3rd operation. With the gestalt of this operation, 
parallax distribution of the corresponding points between right-and-left camera images is used 
as primary description data, using a compound eye image pick-up system (stereoscopic camera) 
as an image input means. Stereo image mensuration is shown in "three dimensional image 
measurement" (Inokuchi, the Sato work, Shokodo) etc. Moreover, he is trying for field area with 
the homogeneous parallax of the photographic subject which should be extracted to serve as 
max by establishing the image pick-up device control means 10, and controlling the angle of 
convergence of a compound eye camera, a focal distance, the base length, etc. by the gestalt of 
this operation based on the parallax data of the parallax extract means 2. While an operator 
observes the label image on the compound eye image display means 8, it may be made to 
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perform this control. Since parallax is acquired only about the field which overlapped between 
right-and-left images, the display of the candidate field after dividing into the field where a 
parallax value is homogeneous is performed only in a duplication field. Field growth is performed 
in the image selection section 1 1 by choosing one candidate for an extract among right-and-left 
images. Moreover, although the hue is used about the secondary description here, it is not 
limited to this. 

[0024] Drawing 4 shows the gestalt of the 4th operation. By obtaining a stereo image as time 
series image data, concomitant use with the gestalt of the 2nd operation is possible for the 
gestalt of implementation of the above 3rd. Drawing 4 shows the example of a system 
configuration. Here, the primary description data extraction means 2 extracts a motion vector 
and parallax distribution, and each description data performs clustering based on whenever 
[ homogeneity ] with field division and the label attachment means 5a and 5b. An input image is 
shown in drawing 8 (a), and the field division result based on the primary description (parallax and 
motion vector) is shown in this drawing (b). 

[0025] Next, the label image by which the deflection between representation characteristic 
quantity (parallax, motion vector) was computed, field division was carried out based on the 
description data with larger deflection, and the area size after division was generated about the 
cluster beyond a predetermined value with the deflection calculation means 12a and 12b is used 
in the case of the candidate field selection for an extract, the distance with the body almost 
same [ this ] from the compound eye animation image pick-up means 1 in a screen — being 
distributed (the deflection of parallax being smallness), when the photographic subject which 
should be extracted has a remarkable motion to a perimeter the direction of the field division 
using a motion vector — reliability — high — moreover, distance with a perimeter and a 
background — the distance of the compound eye animation image pick-up means 1 and an 
object — comparing — enough — being large (the deflection of parallax being size), when all do 
not show a remarkable motion relatively (the deflection of a motion vector is smallness) It is 
because the field division based on parallax is desirable. 

[0026] Drawing 8 (c) shows a processing result, chooses the largest object of a parallax value 
here, and shows the result of having performed field growth using the secondary description 
minute amount (color component). Thereby, the image data which was suitable for the image 
extract according to the size of the existence of a motion of an object and dispersion of 
distance distribution can be used properly. Moreover, you choose the classification of the 
description data beforehand used based on the primary description data, and may also narrow 
down image pick-up modes (a still picture and a stereo image pick-up, or an animation, a 
monocular vision image pick-up, etc.). Moreover, in using only the dynamic-image data from one 
camera, it applies processing with the gestalt of the 2nd operation. In addition, as an image input 
means, it may not be limited to a compound eye image pick-up system, but the image with 
parallax based on a three-dimensions geometric model generated by computer graphics is 
sufficient. 

[0027] Drawing 5 shows the gestalt of the 5th operation. The gestalt of this operation uses an 
input image as the time series image from the image pick-up means 1, such as an ocellus camera 
by which fixed installation was carried out. If it considers as the primary description data, the 
outline field (a location and magnitude) of a part with the primary description data lost motion is 
extracted using inter-frame subtraction-image data. An image is extracted with the application 
of the field grown method based on the selection by label attachment, the display means 8, 
mouse, or the BOINTINGU means 9 of a candidate field, and the secondary description over a 
candidate field. 

[0028] drawing 5 — setting — difference — the primary description data obtained by making 
binary the subtraction image obtained by data extraction means 2a with a predetermined 
threshold with binary-ized means 2b carry out connection processings of them, such as etc., the 
distance between segments is below the number of predetermined pixels — in accordance with 
fixed criteria, without expressing coarse division of a motion vector to a quiescence background, 
therefore evaluating whenever [ homogeneity ]. It carries out label attachment, using as a 
candidate field the field across which the description data (non-zero data) existed the front and 
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behind objective, and carried out grouping, respectively, and they faced, and it leaves as a 
division field. [ which were typically moved in the predetermined direction ] Furthermore, the field 
below predetermined size is omitted. Although the hue is used like the gestalt of each operation 
as secondary description data, it is not limited to this. 

[0029] Drawing 6 shows the gestalt of the 6th operation, the gestalt of this operation — an input 
image — receiving — the primary description data (a motion vector — ) of multiplex resolving 
and a hierarchical type Extract processings, such as parallax, are performed and it is low 
resolution data (for example, after forming an image into low resolving by infanticide processing 
or equalization) as a primary description. The secondary description uses the high resolution 
data of the same classification, such as computing the motion vector for every block by 
detecting a motion vector or dividing image data into the large block of size. According to the 
gestalt of this operation, the high-speed processing by one feature-extraction processing 
algorithm is possible. 

[0030] Drawing 6 shows the system configuration at the time of using a motion vector as primary 
description data. Minute logging is performed by performing a setup (growth nucleus setting 
means 72) of a field growth nucleus with a low resolution from evaluation (whenever 
[ homogeneity ] evaluation means 3) of whenever [ motion vector homogeneity ], and performing 
field growth (field growth means 73) with high resolution. Like the gestalt of the 2nd operation, it 
cannot be overemphasized that field growth means the merge or separation processing of 
subregion based on whenever [ homogeneity ]. Since especially growth nuclei are gestalt ****** 
of this operation, and a field fundamentally represented with the same motion vector, as 
compared with the field growth using a color component, a growth nucleus setup is simplified 
very much, and more nearly high-speed processing is possible for them also in field growth 
processing. 
[0031] 

[Effect of the Invention] As explained above, according to the 1st invention, the image for 
specification in the background of arbitration can be automatically extracted only by directing a 
rough location. Moreover, according to the 2nd invention, an object image can be efficiently 
extracted by utilizing the feature parameter of classification with which plurality differs. 
Moreover, according to the 3rd invention, when especially the description distribution data 
present some homogeneous fields and make them selectable, automatic extracting of the same 
image as the 1st invention can be performed. According to the 4th invention, the same image 
logging as the 2nd invention can be performed, without performing evaluation of whenever 
[ homogeneity ]. 

[0032] According to the 5th invention, an operator can perform easily assignment of the field 
which should be extracted. According to the 6th invention, a setup of the image candidate field 
which should be extracted is automatable. According to the 7th invention, narrowing down and a 
setup of an image candidate field which should be extracted are automatable. 

[0033] According to the 8th invention, the extract and narrowing down of a candidate field based 
on the target motion vector component can be performed. According to the 9th invention, the 
object image candidate field from the field where the motion was detected can be extracted, and 
the same effectiveness as the 8th invention can be acquired. According to the 10th invention, 
narrowing down of an image candidate field and a setup are automatable from a stereo image 
with parallax. According to the 1 1th invention, the operability at the time of only rough directions 
performing the alternative extract of an object domain image can be raised. 
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